Mechanism for streaming media data over wideband wireless networks

ABSTRACT

A system is disclosed having a transmitter to Tirelessly transmit multi-media data. The transmitter includes a module to packetize phase units of a multimedia frame that is to be transmitted. The system also includes a receiver to receive the multimedia frame received from the transmitter. The receiver includes a post processing module to perform error concealment on the phase units.

This non provisional application claims priority to and incorporates byreference the provisional application Ser. No. 60/785,361, filed on Mar.22, 2006 and claims priority thereof.

FIELD OF INVENTION An embodiment of the invention relates to wirelesscommunication, and more specifically, to the transfer of multi-mediadata. BACKGROUND

Due to its voluminous size, multimedia data (e.g., high-resolutionimages, video, high-definition TV) is extremely demanding of bandwidthresources. Thus, there is a critical need to compress multimedia data tofacilitate delivery over bandwidth-constrained transmission mediums.Currently, there are various mechanisms that implement mediacompression.

In particular, there are there are several commercial standards forimage and video compression, including JPEG, JPEG-2000, and JPEG-LS forimage compression, and MPEG-1,2,4, and H.26x family of standards forvideo compression. Both lossless (e.g., JPEG-LS) and lossy (e.g., JPEG,MPEG) are covered under these standards.

When transmitting multimedia data over noisy transmission mediums suchas wireless networks, there is often a need for both compression (tosatisfy the bandwidth constraints) and robustness (to overcome the noisytransmission medium). The latter is classically accomplished through theuse of forward error correction (FEC) methods (such as Reed-Solomoncodes, turbo codes, LDPC codes, etc.) and/or through retransmission oflost packets in case there is (as in TCP/IP networks). Retransmissionschemes ensure reliability. However, retransmission occurs at the costof delays and buffering requirements at both transmitter and receiver,which may be prohibitive or even unacceptable in many applications.

Even if compression is not employed, optimizing the transmissiondelivery for the delivery of video formats is critical for achievingsuperior performance, but this is not included in existing source andchannel coding technology.

Thus, conventional mechanisms in the field of media transmission overwireless networks involve a separation of the tasks of source coding(using state-of-the-art compression technology like H.264 and MPEG-2),channel coding (using state-of-the-art methods like LDPC and turbocodes), or retransmitting lost data in the presence of channel feedback.Such schemes are not a good fit for low-complexity and low-latencyapplications.

SUMMARY

A method, system and apparatus for streaming media data over widebandwireless networks are described. In one embodiment, the system comprisesa transmitter and a receiver, where the transmitter comprises aphased-array antenna to wirelessly transmit media data using beamformingand a first module coupled to the phased-array antenna to packetizephases of media data and to determine a prioritized transmission orderto transmit the packetized phases using the phased-array antenna. Thereceiver receives the packetized phases from the transmitter and has apost processing module to perform error concealment on data in receivedpacketized phases.

DESCRIPTION OF THE DRAWINGS

The invention may be best understood by referring to the followingdescription and accompanying drawings that are used to illustrateembodiments of the invention. In the drawings:

FIG. 1 is a block diagram of one embodiment of a communication system;

FIG. 2 is a more detailed block diagram of one embodiment of thecommunication system;

FIG. 3 illustrates one embodiment of an image frame; and

FIG. 4 illustrates another embodiment of an image frame.

FIG. 5 illustrates more detailed embodiment of the partitioning andpartition scheme.

FIG. 6 illustrates an example a table stored in memory of thetransmitter to keep track of which partitions have been sent for eacharea of the screen.

DETAILED DESCRIPTION

A mechanism providing the robust, efficient and low-latency delivery ofmedia data over wideband wireless networks is disclosed. According toone an embodiment, compression, data partitioning, and post-processing(e.g., spatio-temporal interpolation and error-concealment) mechanismsare combined. In a further embodiment, the combined application layermechanisms are coupled with physical layer components, such asmulti-antenna beamforming patterns in multiple-input multiple-output orMIMO wireless systems, as well as physical layer Forward ErrorCorrection (FEC) code strengths that are matched to Quality of Service(QoS) parameters of the—layer media content.

In the following description, numerous details are set forth. It will beapparent, however, to one skilled in the art that embodiments of thepresent invention may be practiced without these specific details. Inother instances, well-known structures, devices, and techniques have notbeen shown in detail, in order to avoid obscuring the understanding ofthe description. The description is thus to be regarded as illustrativeinstead of limiting.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least an embodimentof the invention. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment.

In the following description, numerous details are set forth to providea more thorough explanation of the present invention. It will beapparent, however, to one skilled in the art, that the present inventionmay be practiced without these specific details. In other instances,well-known structures and devices are shown in block diagram form,rather than in detail, in order to avoid obscuring the presentinvention.

Some portions of the detailed descriptions which follow are presented interms of algorithms and symbolic representations of operations on databits within a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The present invention also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, andmagnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any typeof media suitable for storing electronic instructions, and each coupledto a computer system bus.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear from the description below.In addition, the present invention is not described with reference toany particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof the invention as described herein.

A machine-readable medium includes any mechanism for storing ortransmitting information in a form readable by a machine (e.g., acomputer). For example, a machine-readable medium includes read onlymemory (“ROM”); random access memory (“RAM”); magnetic disk storagemedia; optical storage media; flash memory devices; electrical, optical,acoustical or other form of propagated signals (e.g., carrier waves,infrared signals, digital signals, etc.); etc.

An Example of a Communication System

FIG. 1 is a block diagram of one embodiment of a communication system.Referring to FIG. 1, the system comprises media receiver 100, a mediareceiver interface 102, a transmitting device 140, a receiving device141, a media player interface 113, a media player 114 and a display 115.

Media receiver 100 receives content from a source (not shown). In oneembodiment, media receiver 100 comprises a set top box. The content mayinclude baseband digital video, such as, for example, but not limitedto, content adhering to the HDMI or DVI standards. In such a case, mediareceiver 100 may include a transmitter (e.g., an HDMI transmitter) toforward the received content.

Media receiver 101 sends content 101 to transmitter device 140 via mediareceiver interface 102. In one embodiment, media receiver interface 102includes logic that converts content 101 into HDMI content. In such acase, media receiver interface 102 may include an HDMI plug and content101 is sent via a wired connection; however, the transfer could occurthrough a wireless connection. In another embodiment, content 101includes DVI content.

In one embodiment, the transfer of content 101 between media receiverinterface 102 and transmitter device 140 occurs over a wired connection;however, the transfer could occur through a wireless connection.

Transmitter device 140 wirelessly transfers information to receiverdevice 141 using two wireless connections. One of the wirelessconnections is through a phased array antenna with adaptive beamforming.The other wireless connection is via wireless communications channel107, referred to herein as the back channel. In one embodiment, wirelesscommunications channel 107 is unidirectional. In an alternativeembodiment, wireless communications channel 107 is bi-directional.

Receiver device 141 transfers the content received from transmitterdevice 140 to media player 114 via media player interface 113. In oneembodiment, the transfer of the content between receiver device 141 andmedia player interface 113 occurs through a wired connection; however,the transfer could occur through a wireless connection. In oneembodiment, media player interface 113 comprises an HDMI plug.Similarly, the transfer of the content between media player interface113 and media player 114 occurs through a wired connection; however, thetransfer could occur through a wireless connection.

Media player 114 causes the content to be played on display 115. In oneembodiment, the content is HDMI content and media player 114 transferthe media content to display via a wired connection; however, thetransfer could occur through a wireless connection. Display 115 maycomprise a plasma display, an LCD), a CRT, etc.

Note that the system in FIG. 1 may be altered to include a DVDplayer/recorder in place of a DVD player/recorder to receive, and playand/or record the content.

In one embodiment, transmitter 140 and media receiver interface 102 arepart of media receiver 100. Similarly, in one embodiment, receiver 140,media player interface 113, and media player 114 are all part of thesame device. In an alternative embodiment, receiver 140, media playerinterface 113, media player 114, and display 115 are all part of thedisplay.

In one embodiment, transmitter device 140 comprises a processor 103, anoptional baseband processing component 104, a phased array antenna 105 awireless communication channel interface 106, and apartitioning/compression module 108. Phased array antenna 105 comprisesa radio frequency (RF) transmitter having a digitally controlled phasedarray antenna coupled to and controlled by processor 103 to transmitcontent to receiver device 141 using adaptive beamforming.

In one embodiment, receiver device 141 comprises a post processingmodule 116, a processor 112, an optional baseband processing component111, a phased array antenna 110, and a wireless communication channelinterface 109. Phased array antenna 110 comprises a radio frequency (RF)transmitter having a digitally controlled phased array antenna coupledto and controlled by processor 112 to receive content from transmitterdevice 140 using adaptive beamforming.

In one embodiment, processor 103 generates baseband signals that areprocessed by baseband signal processing 104 prior to being wirelesslytransmitted by phased array antenna 105. In such a case, receiver device141 includes baseband signal processing to convert analog signalsreceived by phased array antenna 110 into baseband signals forprocessing by processor 112. In one embodiment, the baseband signals areorthogonal frequency division multiplex (OFDM) signals.

In one embodiment, transmitter device 140 and/or receiver device 141 arepart of separate transceivers.

Transmitter device 140 and receiver device 141 perform wirelesscommunication using phased array antenna with adaptive beamforming thatallows beam steering. Beamforming is well known in the art. In oneembodiment, processor 103 sends digital control information to phasedarray antenna 105 to indicate an amount to shift one or more phaseshifters in phased array antenna 105 to steer a beam formed thereby in amanner well-known in the art. Processor 112 uses digital controlinformation as well to control phased array antenna I 10. The digitalcontrol information is sent using control channel 121 in transmitterdevice 140 and control channel 122 in receiver device 141. In oneembodiment, the digital control information comprises a set ofcoefficients. In one embodiment, each of processors 103 and 112comprises a digital signal processor.

Wireless communication link interface 106 is coupled to processor 103and provides an interface between wireless communication link 107 andprocessor 103 to communicate antenna information relating to the use ofthe phased array antenna and to communicate information to facilitateplaying the content at another location. In one embodiment, theinformation transferred between transmitter device 140 and receiverdevice 141 to facilitate playing the content includes encryption keyssent from processor 103 to processor 112 of receiver device 141 and oneor more acknowledgments from processor 112 of receiver device 141 toprocessor 103 of transmitter device 140.

Wireless communication link 107 also transfers antenna informationbetween transmitter device 140 and receiver device 141. Duringinitialization of the phased array antennas 105 and 110, wirelesscommunication link 107 transfers information to enable processor 103 toselect a direction for the phased array antenna 105. In one embodiment,the information includes, but is not limited to, antenna locationinformation and performance information corresponding to the antennalocation, such as one or more pairs of data that include the position ofphased array antenna 110 and the signal strength of the channel for thatantenna position. In another embodiment, the information includes, butis not limited to, information sent by processor 112 to processor 103 toenable processor 103 to determine which portions of phased array antenna105 to use to transfer content.

When the phased array antennas 105 and 110 are operating in a modeduring which they may transfer content (e.g., HDMI content), wirelesscommunication link 107 transfers an indication of the status ofcommunication path from the processor 112 of receiver device 141. Theindication of the status of communication comprises an indication fromprocessor 112 that prompts processor 103 to steer the beam in anotherdirection (e.g., to another channel). Such prompting may occur inresponse to interference with transmission of portions of the content.The information may specify one or more alternative channels thatprocessor 103 may use.

In one embodiment, the antenna information includes information sent byprocessor 112 to specify a location to which receiver device 141 is todirect phased array antenna 110. This may be useful duringinitialization when transmitter device 140 is telling receiver device141 where to position its antenna so that signal quality measurementscan be made to identify the best channels. The position specified may bean exact location or may be a relative location such as, for example,the next location in a predetermined location order being followed bytransmitter device 140 and receiver device 141.

In one embodiment, wireless communications link 107 transfersinformation from receiver device 141 to transmitter device 140specifying antenna characteristics of phased array antenna 110 or viceversa.

An Example of a Transceiver Architecture

FIG. 2 is a block diagram of one embodiment of an adaptive beam formingmultiple antenna radio system including transmitter device 140 andreceiver device 141 of FIG. 1. Transceiver 200 includes multipleindependent transmit and receive chains. Transceiver 200 performs phasedarray beam forming using a phased array that takes an identical RFsignal and shifts the phase for one or more antenna elements in thearray to achieve beam steering.

Referring to FIG. 2, Digital Signal Processor (DSP) 201 formats thecontent and generates real time baseband signals. DSP 201 may providemodulation, FEC coding, packet assembly, interleaving and automatic gaincontrol. For purposes herein, a digital signal processor can refereither to a programmable processor which performs digital signalprocessing, fixed computational units which perform the digital signalprocessing, or any combination thereof.

DSP 201 then forwards the baseband signals to be modulated and sent outon the RF portion of the transmitter. In one embodiment, the content ismodulated into OFDM signals in a manner well known in the art.

Digital-to-analog converter (DAC) 202 receives the digital signalsoutput from DSP 201 and converts them to analog signals. In oneembodiment, the signals output from DAC 202 are between 0 and about 1250MHz signals and DAC 202 represents a pair of digital-to-analogconverters for quadrature (I/Q) output.

Mixer 203 receives signals output from DAC 202 and combines them with asignal from a local oscillator (LO) 204. The signals output from mixer203 are at an intermediate frequency. In one embodiment, theintermediate frequency is between 10-14 GHz.

Multiple phase shifters 205 _(0-N) receive the output from mixer 203. Ade-multiplexer is included to control which phase shifters receive thesignals. In one embodiment, these phase shifters are quantized phaseshifters. In an alternative embodiment, the phase shifters may bereplaced by complex multipliers. In one embodiment, DSP 201 alsocontrols, via control channel 208, the phase and magnitude of thecurrents in each of the antenna elements in phased array antenna 220 toproduce a desired beam pattern in a manner well-known in the art. Inother words, DSP 201 controls the phase shifters 205 _(0-N) of phasedarray antenna 220 to produce the desired pattern.

Each of phase shifters 205 _(0-N) produce an output that is sent to oneof power amplifiers 206 _(0-N), which amplify the signal. The amplifiedsignals are sent to antenna array 207 which has multiple antennaelements 207 _(0-N). In one embodiment, the signals transmitted fromantennas ²⁰⁷ _(0-N) are radio frequency signals between 56-64 GHz. Thus,multiple beams are output from phased array antenna 220.

With respect to the receiver, antennas 210 _(0-N) receive the wirelesstransmissions from antennas 207 _(0-N) and provide them to phaseshifters 211 _(0-N). As discussed above, in one embodiment, phaseshifters 211 _(0-N) comprise quantized phase shifters. Alternatively,phase shifters 211 _(0-N) may be replaced by complex multipliers. Phaseshifters 211 _(0-N) receive the signals from antennas 210 _(0-N), whichare combined to form a single line feed output. In one embodiment, amultiplexer is used to combine the signals from the different elementsand output the single feed line. The output of phase shifters 211 _(0-N)is input to intermediate frequency (IF) amplifier 212, which reduces thefrequency of the signal to an intermediate frequency. In one embodiment,the intermediate frequency is between 2-9 GHz.

Mixer 213 receives the output of the IF amplifier 212 and combines itwith a signal from LO 214 in a manner well-known in the art. In oneembodiment, the output of mixer 213 is a signal in the range of about0-1250 MHz. In one embodiment, there are I and Q signals for eachchannel.

Analog-to-digital converter (ADC) 215 receives the output of mixer 213and converts it to digital form. In one embodiment, the signals outputfrom ADC 215 are between 0 and about 1250 MHz signals and ADC 215represents a pair of analog-to-digital converters for quadrature (I/Q)digitization. The digital output from ADC 215 is received by DSP 216.DSP 216 restores the amplitude and phase of the signal. DSPs 211 mayprovide demodulation, packet disassembly, de-interleaving and automaticgain control.

In one embodiment, each of the transceivers includes a controllingmicroprocessor that sets up control information for DSP. The controllingmicroprocessor may be on the same die as the DSP.

DSP-Controlled Adaptive Beam Forming

In one embodiment, the DSPs implement an adaptive algorithm with thebeam forming weights being implemented in hardware. That is, thetransmitter and receiver work together to perform the beam forming in RFfrequency using digitally controlled analog phase shifters; however, inan alternative embodiment, the beamforming is performed in IF. Phaseshifters ²⁰⁵ _(0-N) and 211 _(0-N) are controlled via control channel208 and control channel 217, respectfully, via their respective DSPs ina manner well known in the art. For example, DSP 201 controls phaseshifters 105 _(0-N) to have the transmitter perform adaptive beamformingto steer the beam while DSP 211 controls phase shifters 211 _(0-N) todirect antenna elements to receive the wireless transmission fromantenna elements and combine the signals from different elements to forma single line feed output. In one embodiment, a multiplexer is used tocombine the signals from the different elements and output the singlefeed line.

DSP 201 performs the beam steering by pulsing, or energizing, theappropriate phase shifter connected to each antenna element. The pulsingalgorithm under DSP 201 controls the phase and gain of each element.

The adaptive beam forming antenna is used to avoid interferingobstructions. By adapting the beam forming and steering the beam, thecommunication can occur by avoiding obstructions which may prevent orinterfere with the wireless transmissions between the transmitter andthe receiver.

In one embodiment, with respect to the adaptive beamforming antennas,they have three phases of operations. The three phases of operations arethe training phase, a searching phase, and a tracking phase. Thetraining phase and searching phase occur during initialization. Thetraining phase determines the channel profile with predeterminedsequences of spatial patterns {A_(î)} and {B_(ĵ)}. The searching phasecomputes a list of candidate spatial patterns {A_(î)}, {B_(ĵ)} andselects a prime candidate {A_({circumflex over (0)}),B_({circumflex over (0)})} for use in the data transmission between thetransmitter of one transceiver and the receiver of another. The trackingphase keeps track of the strength of the candidate list. When the primecandidate is obstructed, the next pair of spatial patterns is selectedfor use.

In one embodiment, during the training phase, the transmitter sends outa sequence of spatial patterns {A_(î)}. For each spatial pattern{A_(î)}, the receiver projects the received signal onto another sequenceof patterns {B_(ĵ)}. As a result of the projection, a channel profile isobtained over the pair {A_(î)}, {B_(ĵ)}.

In one embodiment, an exhaustive training is performed between thetransmitter and the receiver in which the antenna of the receiver ispositioned at all locations and the transmitter sending multiple spatialpatterns. Exhaustive training Is well-known in the art. In this case, Mtransmit spatial patterns are transmitted by the transmitter and Nreceived spatial patterns are received by the receiver to form an N by Mchannel matrix. Thus, the transmitter goes through a pattern of transmitsectors and the receiver searches to find the strongest signal for thattransmission. Then the transmitter moves to the next sector. At the endof the exhaustive search process, a ranking of all the positions of thetransmitter and the receiver and the signals strengths of the channel atthose positions has been obtained. The information is maintained aspairs of positions of where the antennas are pointed and signalstrengths of the channels. The list may be used to steer the antennabeam in case of interference.

In an alternative embodiment, bi-section training is used in which thespace is divided in successively narrow sections with orthogonal antennapatterns being sent to obtain a channel profile.

Assuming DSP 101 is in a stable state and the direction the antennashould point is already determined. In the nominal state, the DSP willhave a set of coefficients that it sends the phase shifters. Thecoefficients indicate the amount of phase the phase shifter is to shiftthe signal for its corresponding antennas. For example, DSP 101 sends aset of digital control information to the phase shifters that indicatethe different phase shifters are to shift different amounts, e.g., shift30 degrees, shift 45 degrees, shift 90 degrees, shift 180 degrees, etc.Thus, the signal that goes to that antenna element will be shifted by acertain number of degrees of phase. The end result of shifting, forexample, 16, 34, 32, 64 elements in the array by different amountsenables the antenna to be steered in a direction that provides the mostsensitive reception location for the receiving antenna. That is, thecomposite set of shifts over the entire antenna array provides theability to stir where the most sensitive point of the antenna ispointing over the hemisphere.

Note that in one embodiment the appropriate connection between thetransmitter and the receiver may not be a direct path from thetransmitter to the receiver. For example, the most appropriate path maybe to bounce off the ceiling.

In one embodiment, when the transmission path being currently usedbecomes obscured, another path is searched for using a beam searchprocedure. While this can obtain an improved beam, it can also result ina temporary cessation of communication while the beam search procedureis occurring. Additionally, the communication may have degraded prior tothe beam search or beam tracking operations. Thus, the wireless channelcan experience outages due to the adaptive beam forming. The techniquesdescribed herein for media streaming are used for mitigating the impactof these outages.

An embodiment used for more information on adaptive beam forming isdescribed in U.S. Application Ser. No. 11/706,711, entitled “AdaptiveBeam-Steering Methods to Maximize Wireless Link Budget and ReduceDelay-Spread Using Multiple Transmit and Receive Antennas,” filed onFeb. 13, 2007.

Partitioning and Post Processing

As discussed above, transmitter 140 and receiver 141 include modules 108and 116, respectively. Partitioning module 108 packetizes each phase(partition) of the image-video data. In one embodiment, module 108performs joint compression and packetization of video/image dataoptimized for the wireless streaming of the media. Compression refers toa process of removing redundancy inherent in most media data. Losslessimage coding methods such as JPEG-LS derive their compression efficiencythrough a multitude of concepts. In another embodiment, module 108performs packetization of the video/image data in its uncompressed formoptimized for the wireless streaming of the media. Packetization refersto the packaging of the compressed (or non-compressed) bit-stream (thephases, or partitions) for transmission over a (noisy) transmissionmedium or network. Generally, if conditions of the transmission mediumare not good, fewer partitions are sent. Then, resolution adjustment canbe used to compensate for the conditions.

Delivery of media steaming, especially for high-definition formats,requires large buffers since the transmitters need to buffer incomingstreaming data during control signaling periods, multiplexing formultiple streams, or retransmission due to bad channel conditions. Witha limited buffer, overflows severely damage the media quality.Transmitter buffer management in partitioning module 108 can maintainhigh media quality with a limited buffer by taking advantage ofpartitioning and receiver error concealment. When the buffer cannotaccommodate the incoming packets, the buffer management proactivelydrops some media packets with a low priority in the buffer and leavesenough space for the next incoming packets. The proactive dropping canavoid error concealment failures resulting from data loss due tooverflows and improves the efficiency of buffer use without noticeablemedia quality degradation. When the channel transmission capacity islower than the media data rate, this mechanism is equivalent to sourcecoding rate control.

Post processing module 116 performs actions in order to enhance thedelivered quality at the display unit. Examples include errorconcealment of lost pixels through classical image processingmethodologies such as nearest neighbor copying or pixel-domaininterpolation (that can be spatial, temporal, or spatio-temporal) thatcan be arbitrarily sophisticated by leveraging the latest advances inimage and video processing and noise-removal methodologies. In oneembodiment, using the post processing module 116, the receiver is ableto handle the data that the transmitter sends it, even if it is lessthan all the partitions through error or loss concealment.

If feedback from the receiver to the transmitter of which data isreceived and/or received correctly is available, then this can be usedboth to facilitate retransmissions as well as optimization of the choiceof data to be transferred to further improve the performance ofmulti-media optimized streaming. If the receiver informs the transmitterwhich packets are correctly received, the transmitter and receiver canbe in sync.

Packetization/Compression

Packetization/compression module 108 allows for the compression and thepacketization approaches at the application layer to be matched to eachother and jointly made aware to receiver 141 via post-processing module116. FIG. 3 illustrates one embodiment of an image frame forillustrating post-processing coupling. In this embodiment, the imageframe is tiled into a “checkerboard” pattern having two phases(partitions) “A” and “B”.

In one embodiment, the size of the phase units depicted as A and Brespectively can be selected as single pixels or small blocks. Forexample, the units corresponding to A and B may include single imagepixels or arbitrarily sized image blocks. In other embodiments, theimage frame could be extended to additional phases. For instance, FIG. 4illustrates the image frame tiled into a “checkerboard” pattern havingfour phases “A”, “B”, “C” and “D”.

In one embodiment, each of the phases are separately packetized (andcompressed) at module 108 in order to control a trade-off betweencompression (e.g., maximized if image not separated into multiplephases) and robustness (e.g., leveraging diversity gains of receiving atleast one phase correctly). Particularly, compression efficiency ispotentially compromised by breaking up the image into its phases due toa more diluted neighborhood context that can be used for prediction.That is, the correlation stricture of the pixels transmitted under phaseA independently is weaker than that in the composite case oftransmitting phases A and B jointly.

However, the separate phase compression and packetization results inincreased diversity or robustness. For instance, if certain pixels inthe B phase are lost (e.g., due to intermittent losses induced in thebeamformed wireless system that can have interruptions in channelusage), the pixels can be efficiently interpolated at the receiver fromthe neighboring pixels in phase A. Thus, the system may be adjusted totrade off compression efficiency for robustness, with a diversity gaininduced by the number of phases.

According to one embodiment, sub-sampling ratios of an image frame maybe further be tailored in the case of color images and image sequences,to indicate whether the image frame is a luminance frame or achrominance frame (e.g., Y—Cr-Cb format). In such an embodiment, thechrominance components are sub-sampled more aggressively since thesecomponents are more predictable.

In a further embodiment, compression/packetization module 108 implementsa prioritized transmission order to transmit and packetize the mediacontent. In this embodiment, phase A of the image frame in FIG. 3 isprioritized over phase B. Thus, phase A is transmitted with maximalreliability, while phase B is transmitted if there is sufficientbandwidth and/or time depending on the latency constraint.

Adaptive transmission order can be implemented in a phase (partition)scheduler in partitioning module 108 that is in charge of determiningthe serving order of those phase (partition) queues in which thecorresponding phases (partitions) are stored. The scheduler firstcalculates the scheduling weight for each head-of-the-line phase(partition) packet based on its phase priority, delay, play deadline,queue length, and the current channel conditions, and then chooses thehead-of-the-line packet with the largest weight to serve. The weightcalculation is to maximize the media quality and robustness. Forexample, a higher delay results in a higher weight, and with bad channelconditions the scheduler lowers some phase (partition) queues' priorityto fit the channel capacity.

In case phase B is not received, en-or-concealment strategies (discussedin more detail below) can be invoked at post processing module 116.Accordingly, it is possible to ensure that the signal reconstructionquality at receiver 141 would be minimally compromised due to thesuperior interpolation possibilities that leverage the presence of phaseA. In uncompressed partitioning systems, it is possible to reconstruct aregion of the image at some quality with any one or more of the phasespresent. All of the partition dropping and transmission orderingtechniques described herein can apply equally well to the uncompressedas compressed cases.

The prioritized transmission order minimally affects the overallcompression efficiency. For example, while the compression efficiency ofphase A may somewhat be effected as compared to an embodiment with noprioritized transmission, the loss is recovered by compressing phase Bmore efficiently. This is done by compressing phase B by leveraging thepresence of the higher priority phase A samples at receiver 141.

For example, in the presence of a feedback link, it is possible toguarantee that no phase B packets for a video frame is transmittedunless all of the phase A packets have been reliably delivered, therebyensuring the presence of phase A samples at receiver 141. Phase B datacan subsequently be compressed more efficiently than a classical JPEG-LSstyle compression framework that exploits only a causal predictionneighborhood. Due to the presence of a richer non-causal predictioncontext, the phase B data can be compressed extremely efficiently. Thus,prioritization results in a highly efficient transmission order thatcombines efficiency with robustness by leveraging the image structure atreceiver 141.

The above-described prioritization schemes are relevant not only due toa fluctuating channel capacity pipe (due to the transmissionenvironment) but also due to the variable bit rate nature of naturalvideo sequences, as well as a demand to break up image frames intosmaller sizes for packet transmission. That is, there can be a widevariance in the bit rate requirements for video frames (e.g., typicalvideo sequences comprise “high action” and “low action” frames as wellas high-action and background components in each frame). In such cases,it may be expensive or infeasible to “smooth out” the potentially largevariations of the frame bit rate requirements, as the bufferingrequirements as well as the induced delays could be unacceptable. Insuch scenarios, there may be scenarios where it is not possible to sendall the data, and prioritization choices have to be made.

The above-described prioritization mechanism performed by module 108 maybe further coupled with physical layer attributes. For example, inbeamforming-based wireless transmission systems, there are well-knowntradeoffs at the physical layer between multiplexing gain (deliverablebit rate) and diversity gain (robustness level). In one embodiment,these parameters are matched to the QoS parameters of the content. Forexample, the higher-priority packet stream of phase A can receiveincreased diversity gain, or higher signal power in order to transmit itmore reliably. This is may occur in scenarios involving multi-antennaadaptive beam-forming where there can be interruptions in multi-pathsignal strengths, and periods where the channel does not work very well.In such cases, when the capacity is fluctuating, it is imperative to getthe prioritized phase A data through, and do “best effort” for phase B.

In another embodiment, the prioritization of the application content(QoS levels) can be matched with prioritized signal power allocation inaddition to the diversity-multiplexing tradeoffs of the multi-antennasystem when transmitting higher-priority packet content (e.g., phase Apackets). This would allow for an unequal power allocation at thephysical layer that is efficiently matched to the importance of thecontent being delivered.

In addition to beamforming patterns and varying signal power that ismatched to the QoS level of the content, it is possible to vary thestrength of the physical layer FEC code needed to combat channel loss,in accordance with the priority of the content. For example, in theabove-described prioritized transmission scheme, it is possible to tunethe strength of the FEC code to the QoS parameters of the media content(e.g., higher-priority phase A can get a stronger code).

In embodiments where feedback is available to indicate which packetswere delivered correctly or incorrectly, it is possible to jointlyexploit this information with the mechanisms described above. Forexample, it is possible to prioritize the more important packets to beretransmitted, or to use this feedback information to adapt thescanning, compression, and transmission strategies “on the fly.” Forinstance, phase A packets can be re-transmitted with higher priority(using any or all of the described prioritization methods) over those ofphase B in the face of time deadlines (e.g., the time by which contentmust be output for display on a display device).

In one embodiment in which uncompressed data is being transmitted, everyvideo block receives a fixed number of retransmissions that areprioritized according to the importance of their content (e.g., phase Aversus phase B) in a manner that maximally exploits theerror-concealment strategy at the decoder. That is, retransmissions oflost packets can be executed in a prioritized order that couples the QoSparameters of the content to the error-concealment strategy at thedecoder. In this manner, the likelihood of correctly transmitting thephase A blocks is improved.

In addition to prioritization into multiple signal phases (as in FIGS. 3and 4), other embodiments may alternate multi-resolution decompositionsor coarse-to-fine representations of the media content. For instance, inthe case of FIG. 3, in addition to sending pixels corresponding to phaseA and phase B separately (in the raw pixel domain), it is possible tosend a “coarse” version (e.g., (A+B)/2 which represents a crude average)and a “detail” version (the difference (A−B)) with lower priority. Insuch an embodiment, the coarse version would have a higher priority. Anadvantage of this approach is the possibility of a superiorinterpolation strategy at the receiver due to the presence of a“high-quality” coarse representation of the media. This decompositioncan be done with minimal increase in complexity, possibly increasedcompression efficiency, and improved error-concealment potentialresulting in a superior delivered quality at receiver 141.

In one embodiment, transmitter 140 determines that when losses are highand/or bandwidth supported by the channel is lower, in efforts to assurethat all blocks' higher priority phases (i.e. phase A) patterns getthrough, the transmission of lower priority blocks (i.e. phase B) can besuppressed in part or entirely. Thus, resolution shifting may occur whenthe transmitter 140 encounters a worse channel, and transmitter 140automatically only transmits the phase A blocks and receiver 141 canreconstruct into a lower resolution version. Since receiver 141 isalready designed to handle reconstruction and error concealment whensome of the phases are missing, this resolution shifting can be doneusing the standard error concealment techniques.

Post Processing

According to one embodiment, post processing module 116 interpolates amissing pixel in one phase from pixels in other phases and/or otherpixels in the same phase. For instance, a missing pixel in phase B couldbe interpolated by a simple average of the four abutting pixels in phaseA.

Alternatively, the missing pixel can be interpolated by an average ofthe four diagonally adjacent pixels in phase B if individual pixelserrors can be detected. In a packetized system where entire packets aredetected as present or absent, all of the pixels in phase B can beinterpolated from all of the pixels in phase A for a given block if thepacket including all of the data for the phase B portion of the block isdeemed missing. In scenarios where the phase B data is received butphase A is not, phase A pixels could be reconstructed from phase Bpixels.

Further, error-concealment techniques may be implemented that aretightly coupled with the compression and transmission mechanismsdiscussed above with respect to packetization/compression module 108.For example, in the presence of a multi-resolution, it is possible touse both a spatial context but also generalize this to 3-D concealmentstrategies (e.g., 3-D checkerboards involving spatial and temporaldimensions). For instance, checker-board patterns are retained overtime, where the A and B phases alternate between consecutive frames,allowing for maximal exploitation of the inherent spatio-temporalcorrelation structures in video sequences.

Such sophisticated error-concealment methods may involve excessiveamounts of buffering that may not be feasible in some small-bufferconstraint applications. In such cases, where a limited buffer isavailable, it may be possible to include a “coarse” version of theprevious frame in the temporal interpolation context in order toapproximate the performance of a full-buffer system. For example, it maybe possible to enhance the spatial interpolation schemes of FIGS. 3 and4, with the presence of a coarse temporal frame.

The joint packetization/compression mechanism discussed above allows forvery low complexity and low buffer requirement constraints, making itextremely attractive to a number of application scenarios involvingmedia transmission and streaming over wireless networks and othercommunication mediums characterized by intermittent connectivity, e.g.as found in wireless networks with high degrees of fading, or inmulti-antenna physical layer wireless systems.

FIG. 5 illustrates more detailed embodiment of the partitioning andphase scheme described above. Referring to FIG. 5, incoming data 501 isreceived by the partitioning and compression block 502 to perform AVformatting. In one embodiment, the incoming data arrives in raster scanorder. Note that in other embodiments, partitioning and compressionblock 502 does not perform compression.

Partition unit 503 partitions the incoming data into a number of phases,A-D in this example. The wireless transmitter 505 selects among thephases to transport base on the prioritizations described above. In oneembodiment, the packet scheduler controls the selection and causes thePHY to initiate the transmission of the video data.

Receiver 506 receives the wireless transmission via its PHY and its MACsends the data to undergo AV formatting (concealment/departitioning)using AV formatting block 507 to produce an output on video port 510).

Selection to transmit a particular phase is based on the prioritizedtransmission order described above as well as, in one embodiment,feedback received from the receiver 506 on a feedback channel, such asshown in FIG. 1. Feedback channel allows the receiver to tell thetransmitter what it has received.

In one embodiment, the MAC allows performing reordering of the data tobe transmitted to transmit different streams of the various phases A-D.

The partitioning unit block 502 keeps track of which phases have beensent. In one embodiment, the transmitter buffer management describedabove and the packet scheduler are contained in the partition unit 503.FIG. 6 illustrates an example a table stored in memory of thetransmitter to keep track of which phases have been sent for each areaof the screen. In one embodiment, the screen is divided into sets ofrows. In one embodiment, there are four portions of the display and eachportion has ten rows of lines in it. Using the feedback, the transmitteris able to update the phase status for each area of the screen and basedon what data has been sent and the timing that is required to transmitthe data, the transmitter can decide which phases have priority forsending to the receiver.

In one embodiment, the priority is based on the timing by which thereceiver must send the data to the video port. This time is limited andis usually based on the amount of time the video is displayed on thescreen. The timing between the time at which incoming data is receivedto the time that data is output on the video port is based on timestampsthat are associated with the data that is received. Because of clocksynchronization, the transmitter can determine when the time stamp isable to determine when the receiver is expected to play the packet out.That is, there is only a certain amount of time from the time theincoming video data 501 is received to the time it must be output onvideo port 510. Thus, for each area of the screen, the transmitterobtains an indication of which phases have been sent and the area of thescreens operate as a sliding window such that the phases states aremaintained for the area of the screen until that area of the screen isto be replaced with the next set of data that is to be display on thatportion of the screen. In one embodiment, the prioritization determineswhich phases get transmitted and this may be based on minimizing theminimum number of phases received the by the receiver for any part ofthe screen. In another embodiment, the prioritized transmission is basedon maximizing the minimum number of phases in these parts. As an area isabout to expire, prioritization may be based on that ensuring there is acertain level of coverage of that area of the screen. This can be aproblem in adaptive beam forming systems where there may be interruptionin the beam and after transmission begins again a prioritization iscritical to insure that areas that are about to expire on the screenhave data to be presented. In essence, the older data can cause anurgency to have its partitions sent to the receiver for display.

In one embodiment, rate selection is tied into the urgency for deliveryof data such that, for example, if the channel only supports half thedata rate, then only half the phase data may be sent. That is, based onthe data rate available, the amount of phase data that is transmitted iscontrolled.

Note that as far as the receiver is concerned, its concealment systemallows the receiver to utilize the data it does receive to generate thedisplay. The receiver determines whether it needs to interpolate databased on the data it has received, or take other actions which arenecessary to display the video. In one embodiment, the concealmenttechniques that are used are referred to as edge aware concealmenttechniques.

Also shown in partitioning block 502 is compression block 504 which isoptional. The compression may be performed as described above.

Whereas many alterations and modifications of the present invention willno doubt become apparent to a person of ordinary skill in the art afterhaving read the foregoing description, it is to be understood that anyparticular embodiment shown and described by way of illustration is inno way intended to be considered limiting. Therefore, references todetails of various embodiments are not intended to limit the scope ofthe claims which in themselves recite only those features regarded asessential to the invention.

1. A system comprising: a transmitter comprising a phased-array antennato wirelessly transmit media data using beamforming, and a first modulecoupled to the phased-array antenna to packetize phase units of mediadata and to determine a prioritized transmission order to transmit thepacketized phase units using the phased-array antenna; and a receiver toreceive the packetized phase units from the transmitter, the receiverhaving a post processing module to perform error concealment on data inreceived packetized phase units.
 2. The system defined in claim 1wherein the first module drops media packets having a lower prioritythan the media packets being transmitted with the phased-array antenna.3. The system defined in claim 1 wherein the first module calculates aweight for each packetized phase unit at an output of each of aplurality of queues and selects one packetized phase unit to transmitfrom one of the plurality of queues based on the weight.
 4. The systemdefined in claim 1 wherein the first module selects one packetized phaseunit to transmit from a group of packetized phase units at a head ofeach of a plurality of queues to transmit based on at least oneparameter from a set of parameters.
 5. The system defined in claim 4wherein the set of parameters includes one or more of a group consistingof: phase priority, delay, play deadline for a phase, queue length, andcurrent channel condition.
 6. The system defined in claim 1 wherein thetransmitter further comprises a MAC and a physical layer, the MAC tocontrol the selection of packetized phase units and cause the PHY toinitiate the transmission of the video data.
 7. The system defined inclaim 1 further comprising a unit to keep track of which phase unitshave been sent to the receiver for each area of the screen.
 8. Thesystem defined in claim 7 wherein the receiver sends feedback to thetransmitter regarding received phase units and the unit updates statesfor said each area of the screen based on the feedback.
 9. The system ofclaim 1 wherein the first module jointly compresses and packetizes allunits of a first phase of the media and jointly compresses andpacketizes all units of a second phase.
 10. The system of claim 1wherein the first module prioritizes the first phase units to bepacketized and transmitted prior to the second phase units.
 11. Thesystem of claim 1 wherein no phase units of the second set aretransmitted until all of the phase units in the first set present in thepartition transmission buffers have been transmitted.
 12. The system ofclaim 10 wherein the prioritization of the first set of phase units bythe first module is matched with a prioritized signal power allocated bythe phased array beam forming antenna.
 13. The system of claim 1 whereinthe post processing module interpolates missing pixels in the second setof phase units using data from the first set of phase units.
 14. Thesystem of claim 1 wherein the post processing module interpolatesmissing pixels in the second set of phase units by averaging a diagonalof pixels ill the second set of phase units surrounding each missingpixel.
 15. The system of claim 1 wherein the transmitter furthercomprises: a processor coupled to the first module; a baseband signalprocessor coupled to the processor and the phased array beam formingantenna; and a wireless communication channel interface coupled to theprocessor.
 16. The system of claim 15 further comprising: a mediareceiver; and a media receiver interface coupled between the mediareceiver and the first module.
 17. The system of claim 15 wherein thereceiver further comprises: a processor coupled to the post processingmodule; a baseband signal processor coupled to the processor and thephased array beam forming antenna; and a phased array beam formingantenna coupled to the baseband signal processor.
 18. A transmittercomprising: a phased-array antenna to wirelessly transmit media datausing beamforming, and a first module coupled to the phased-arrayantenna to packetize phase units of media data and to determine aprioritized transmission order to transmit the packetized phase unitsusing the phased-array antenna.
 19. The transmitter defined in claim 18wherein the first module drops media packets having a lower prioritythan the media packets being transmitted with the phased-array antenna.20. The transmitter defined in claim 18 wherein the first modulecalculates a weight for each packetized phase unit at an output of eachof a plurality of queues and selects one packetized phase unit totransmit from one of the plurality of queues based on the weight. 21.The transmitter defined in claim 18 wherein the first module selects onepacketized phase unit to transmit from a group of packetized phase unitsat a head of each of a plurality of queues to transmit based on at leastone parameter from a set of parameters.
 22. The transmitter defined inclaim 21 wherein the set of parameters includes one or more of a groupconsisting of: phase priority, delay, play deadline for a phase, queuelength, and current channel condition.
 23. The transmitter defined inclaim 18 further comprising a MAC and a physical layer, the MAC tocontrol the selection of packetized phase units and cause the PHY toinitiate the transmission of the video data.
 24. The transmitter definedin claim 18 further comprising a unit to keep track of which phase unitshave been sent to the receiver for each area of the screen.